posted 09-20-2006 10:38 AM            

A two page cheat sheet on the Utah Numberical Scoring System.
http://www.raymondnelson.us/qc/Utah_ZCT_Scoring_System.pdf

For anyone not familiar with the Utah Sytem - just download and read this. Its simple, easy to use, and easy to remember. It offers diagnostic accuracy as good or better than other systems, with better reliability.

It is very similar to the new DodPI scoring criteria - the defensible dozen - presented by Don Krapohl at the recent APA conference. Mr. Krapohl recommended the Bell, Raskin, Honts, and Kircher (1999) article as a suggested learning reference.

Barry has reduced the salient information to two pages.

Enjoy,

r

------------------
"Gentlemen, you can't fight in here, this is the war room."
--(from Dr. Strangelove, 1964)

posted 11-04-2006 09:40 AM         

Send Nelson 20 bucks!

posted 11-04-2006 06:04 PM         

If you go giving out the password for free, how is Nelson going to be able to buy his motorcycle stuff?

posted 11-04-2006 09:10 PM         

Duras,

I'll give you the password for $15.00. If you know a guy named Sackett, you might be able to get if for $10.00. Jack O may go as low as $10.00 but you'll have to check with him on that.

posted 11-05-2006 12:09 AM            

r

------------------
"Deterrence is the art of creating in the mind of the enemy the fear to attack."
--(from Dr. Strangelove, 1964)

posted 11-05-2006 03:56 PM         

-13 Deceptive +7 Truthful.

Cleves scoring criteria is basically the same as the Utah system. So why the big difference in required totals ??

posted 11-05-2006 04:32 PM            

Every study to date has shown the Backster scoring system is biased against the truthful. The cutoffs you cite are designed to fix that, but it doesn't happen in real life. What are the cut-offs with five charts as are used (sometimes) on Utah tests? You'll find they're scores few ever achieve. (Utah's don't change. Why?)

The best way that I've seen to discuss this one is to get a bunch of tests in which ground truth is known and have a bunch of people score them individually (using all different systems), and then have everybody score / discuss them as a group. It's a great way to learn what works well and what doesn't, and it's a great way to build confidence in one's ability to score charts.

posted 11-06-2006 02:20 PM         

Let’s take the case of Bennie, a 30-year-old janitor at a junior high school. Bennie is a “Forrest Gump” type of guy – a decent person who lives by the rules. He’s also a mama’s boy. Bennie is not sophisticated. He’s a HS grad; a shy townie. Bennie is accused of exposing his penis to a sexually precocious 13-year-old girl. The girl made the accusation because she doesn’t like the way Bennie looks at her. (She has her own issues.) Authorities pressure Bennie into a specific-issue polygraph with an examiner using the Utah protocol. To Bennie, the mere suggestion that he exposed himself to this girl takes his breath away. Over the course of three charts, Bennie racks up a -5. Nothing overly dramatic, mainly just scattered pneumo reactions. To satisfy the Utah protocol, the examiner runs two more charts. Chart #4 is flat as a pancake. (Perhaps some habituation has set in.) Finally, on the last relevant question of chart 5, Bennie has a distinct apnea. Bingo! Now, according to the Utah rules, Bennie is “scientifically” DI?

Here’s another one…. Suzy is married to Bo, a good ol’ boy who happens to be a textbook example of a DV abuser -- much like “Earl” of Dixie Chicks infamy. Suzy’s next-door neighbor is Fred, a kind and handsome man who is a veterinarian. Suzy adores Fred. She dreams about him. She wants him. Bo sees the “look of love” in his wife’s wandering eyes and gets the notion the two are having an affair. Bo pressures Suzy into a polygraph. Suzy is, and always has been, an honest church-going gal who would never cheat on her husband or even hurt a fly. Suzy’s performance on the polygraph mirrors Bennie’s – that is to say scattered mild reactions, a flat fourth chart, and finally a noticeable apnea on chart five. Again, I ask you: Has Suzy now been “scientifically” shown to be DI?

In either case, the fallout from a false-positive result will be absolutely devastating.

Sure, a thoughful examiner can use the “written statement” technique to ease the stigma of the relevant questions, but an appreciable degree of emotional impact is still going to be there, as both Bennie and Suzy are afraid of a false-positive result.

It seems to me that fixed-threshold tests can be inherently more prone to false-positives (and false-negatives) than increasing-threshold tests.

Barry, you mentioned the “real world.” Based on my experience, I feel that Backster got it right when he devised his increasing-threshold and asymmetrical cutoffs, but it was Jim Matte who came up with a superior variant to the Backster system when he developed the Quadri-Track ZCT. Using the Quadri-Track method, one always scores to the preceding control, thereby eliminating any claim of bias against the truthful examinee. The scoring is asymmetrical: -15/+9 for three charts with three (mandatory) relevants. Most importantly, the Q-T’s fear/hope “track” is invaluable in assessing the likelihood of false positive or false positive elements. Also, the Quadri-Track system has a unique scoring rule that is designed to safeguard against countermeasures. (A description of the Matte Quadri-Track ZCT is summarized elsewhere in this forum.)

The downside? Not much. Matte’s Quadri-Track ZCT takes a little longer to administer, and an examiner must be thoroughly acquainted with its structure in order to prepare the examinee and properly evaluate the charts. But once you get into it, the Quadri-Track ZCT works beautifully.

By the way, Matte’s seminal study of the Quadri-Track ZCT – based on 122 real-life cases – was one of only seven field studies accepted by the NRC/NAS and used in its 2003 report “The Polygraph and Lie Detection.”

The Utah system has gained a lot of momentum, but popularity alone should not dictate what tools we ought to have in our toolbox. Given the public’s perception of polygraphy today (vis-à-vis the problem of false positives and the whole “junk science” debate), as well as the abundance of CM information so easily available, it’s clearly time to become re-acquainted with the Matte Quadri-Track ZCT Technique and its underlying principles.

posted 11-06-2006 03:41 PM            

First of all, what evidence do you have to show the Backster system would fair this person any better? You're comparing apples to oranges. You want to score Backster features vs. Utah features, and then you want to use Utah cut-offs vs. Backster cut-offs.

The hypotheticals you describe sound like the people on the anti sites. They really don't add anything to the discussion. Why? In the end all you've shown is that a certain percentage of tests will be wrong. We already know that.

The question is whether the Backster system helps or hinders avoiding more false positives. All the research says it doesn't.

As far as asymmetrical cutting scores go, I agree with you (and Backster), but I've said that here before. The question is, "What is optimal?" In the research Don and I did, we found the evidentiary scoring rules were best - regardless of what scoring system anybody used. (You will notice, though, Utah's +/-6 results in almost symmetrical results.)

I don't recall off the top of my head, but I think the Utah folks were scoring to the prior CQ before Matte was. Moreover, you can score any CQT with a validated scoring system, which means you can score a Matte test using the Utah scoring system. That means you can ditch Matte's rules and score to the strongest CQ on either side of the RQ, and you can use +/-6 cut-offs. The research shows that results in better accuracy - the opposite of what you state, which is another built-in bias against the truthful. That aside, you seem to think the Utah test will result in more false positives than the Matte rules, which is interesting given Matte's use of such rules as the "Dual-Equal Strong Reaction" resulting in a negative score. (Frankly, it that doesn't show a bias, I don't know what does.)

I was trained in the Matte tests by Jim Matte. For the life of me I can't figure out why he believes his tests have a built-in CM safeguard. To date, he hasn't been able to convince me, and I am not alone there. How does the inside track help you assess a false positive? On what basis can you make that claim? If you get a positive score on the inside track, and a negative on the two RQs, do you now presume a false positive?

Matte's "seminal study"? How about the only study? That's not a plus. One of the criticisms of the NAS report is how they chose their studies. Many do not consider his study to be high-quality, yet they used it. Few would complain as it brings up the average accuracy of the CQT.

I will say this though: I am and always have been suspicious of the study. All those cases and no errors? What are the chances? Why have there been no independent studies since Jim's? How can you and he draw such far-reaching conclusions based on a single study done by the person who has an interest in the outcome?

I've used the Matte test before (Quadri-Track), but only in limited circumstances. It's not a "validated" test, and, for the most part, I run my tests as if they will be used in court, which is problematic with this one.

posted 11-06-2006 04:34 PM            

It would be interesting to have a more articulate discussion about the fear/hope inside track issues. While I agree the whole issue is interesting, I don't find this explanation satisfactory to detect the presence or absence of CMs. I also think it is a little reductionistic to suggest that fear and hope alone (independent of the handful of other mechanisms that are well described in the pschological and physiological literature) begins to account for the distinct but overlapping dichotomies of both deception/truth regarding the target issue and faking/authenticity within the examination.

I like hypotheticals, but they only explain what is possible, not what actually happens. For example, you could flip a quarter 10 times, and its possible that you could get heads more than six or seven times in a row.

Here is my sample.

t h h h h t h t t t

Its just unlikely. Heads four times in a row is within the normal range of probability.

Now flip the quarter 200 times - that's 20 x 10 or 20 samples. (Or is it?) Now what do you think the probability is of obtaining heads six times in a row?

Think about it.

This seems off track, but its not. As experienced examiners, who've conducted a few hundred or few thousand exams, we've all seen the unlikely. It becomes more likely the longer we stay in the field, just as the odds in Vegas get worse the longer you sit at the table. Repeated trials have the effect of increasing, through chance, the probability that we observe something spurious. The mistake is in thinking that we are on to something, without replicating our experiment and without accounting for our observations with consideration for other explanations, other sciences, and the laws of inference.

The mistake is basing a professional opinion or empirical conclusion on a few anecdotes.

There are times that our human desire for understanding and predictability dispell us to infer understanding inaccurately - that is why counterintuitive models are important (and its how casinos make a godzillian dollars off gamblers.)

We're smarter to look at the data, and smarter still to understand the workings of inference that help us to interpret that data.

It would make more sense to have this discussion while looking at a set of confirmed charts that include the inside track. I could host them in a secured area, for some really interesting discussion. (I have a recent confession on an alleged sex assault, but its harder to find confirmed NDI in my work. Plus I don't use the inside track.) That way we could discuss the data points and not just the hypotheticals.

r

------------------
"Deterrence is the art of creating in the mind of the enemy the fear to attack."
--(from Dr. Strangelove, 1964)

posted 11-06-2006 05:47 PM            

I have a good example of where it worked really well. I have others in which the zone was a waste of chart paper.

posted 11-07-2006 10:00 PM            

This isn't a simple task. You are trying to identify a rather abstract variable and say that it alone is the reason for such a hard to believe accuracy rate. I don't know that it's worth the effort.

posted 11-07-2006 10:03 PM            

I'm happy to host, in a secured area, an anonymous set of charts that anyone would be willing to send.

I've posted several of my own tests, to some very interesting discussion. While its a little vulnerable to (metaphorically) pull our own pants down in front of our professional peers, my experience has been quite positive. I'm sure not everyone has agreed with my tests, reports, or my take on a subject, but this forum is a tribute to everyone's ability to remain professional.

I think it would be great to be able to look at some of the current media cases - like those pesky Mike Jones charts. C'mon Kresnik, lesseeum. Or maybe if Tremarco comes to Colorado to test Haggard for free, as offered, we could maybe peek at those charts.

Seriously now. We don't talk much about this kind of thing in polygraph, but social workers are trained in something they call "single-systems research," which is really just a form of case study (but every profession likes to put its own brand on things doesn't it).

If anyone has a set of charts, with the inside track, I'm sure there are a lot of examiners who would benefit from seeing them and learning how this works. You can email me about any kind of chart - I (or you) could strip the examinee and examiner's name and all details.

r

------------------
"Deterrence is the art of creating in the mind of the enemy the fear to attack."
--(from Dr. Strangelove, 1964)

posted 11-08-2006 09:17 AM         

I would love to see some specific issue test posted regardles of the format. It would make for a great exchange. In the past, we have held informal examiner meetings in Northern California for just that reason. The exchange has always been well received. With Nelson's ability to post them in a secure fashion, it just seems like a great idea.

posted 11-08-2006 10:03 AM            

r

------------------
"Deterrence is the art of creating in the mind of the enemy the fear to attack."
--(from Dr. Strangelove, 1964)

posted 11-08-2006 10:34 AM         

What are your format requirements for duplication? Does the software matter?

posted 11-09-2006 12:41 AM            

Easiest would be to print to .pdf or .jpg using the CutePDF or LeadTools ePrint utilities.

r

------------------
"Deterrence is the art of creating in the mind of the enemy the fear to attack."
--(from Dr. Strangelove, 1964)

posted 11-09-2006 09:10 AM         

Sorry for the delay, but I had to keep some ice on those two shiners you gave me in your response to my posting.

Regarding Matte’s Quadri-Track ZCT, Don Krapohl himself told me: “My instincts tell me that it is just as good, but no better, than the other single-issue ZCTs.” Okay,as a relative newcomer (two years in practice), who am I to argue? But personally, I think it’s somewhat “better.” Maybe it’s the asymmetrical, increasing-threshold format coupled with scoring to the preceding control. Maybe it’s the insight afforded by the “inside” (fear/hope) track. Maybe it’s the robust (-15/+9) cut-offs. In any case, after administering a MQTZCT, when I give my report to an attorney, spouse, employer or law enforcement official, I’m highly confident of the result.

>>First of all, what evidence do you have to show the Backster system would fair this person any better? You're comparing apples to oranges. You want to score Backster features vs. Utah features, and then you want to use Utah cut-offs vs. Backster cut-offs.

Not quite. I was talking about the Matte system, not the Backster system. My apologies if that wasn’t clear. But in any event, both scenarios I described would have resulted in “inconclusive” findings -- with the inside track pointing toward a false positive -- which is vastly preferable to an innocent person being branded a liar. A false positive is what would have happened, in my hypotheticals, with the broad-brush approach inherent to a fixed-threshold scoring system in a case with a marginal score achieved over five charts.

>>The hypotheticals you describe sound like the people on the anti sites. They really don't add anything to the discussion. Why? In the end all you've shown is that a certain percentage of tests will be wrong. We already know that.

Is every single individual who posts a beef on an “anti” site a crybaby, pedophile, miscreant, scumbag or other form of ne’er do well? Are there no legitimate grievances due to false positives? We all know that a competent examiner can detect deception/truth at levels far above chance, but it’s those examinees that fall into the false-positive category – the people that you so easily dismiss with the words “…a certain percentage of the tests will be wrong. We already know that.” – that troubles me so greatly. Matte’s Quadri-Track ZCT was designed specifically to help protect against false positives. If there’s such great ballyhoo about false positives in the court of public opinion, why not take a closer look at a technique designed to minimize such consequences?

I’m interested in knowing what your response to the victim of a false positive result would be in the case of a -6 achieved over five charts. “Oops?” “Stuff happens?” “Sorry about that?” “The system I used is shown to be 90% accurate and is endorsed by leading government researchers, but collateral damage is simply a fact of life in this business?” In your pre-test phase, do you tell your examinees that even though your technique (Utah) is “the most researched and supported out there,” there’s still a solid one-in-ten chance that you’ll be flat-out wrong? Do you make it clear it’s the legal/social equivalent of placing a 10-shot revolver, with one round in the cylinder – and let’s make it a .500 S&W loaded with Golden Dot hollow points for better “dramatics” -- to their head, spinning the chamber and pulling the trigger? Now, you can argue that the Utah system is competitive, on a diagnostic level, with MRI, mammography, EEG, or whatever. But there’s one big difference: Should a patient “fail” one of those medical diagnostic procedures, they likely won’t be charged with a crime.

>>The question is whether the Backster system helps or hinders avoiding more false positives. All the research says it doesn't.
Again, I was talking about the Matte system, not Backster.

>>I don't recall off the top of my head, but I think the Utah folks were scoring to the prior CQ before Matte was. Moreover, you can score any CQT with a validated scoring system, which means you can score a Matte test using the Utah scoring system. That means you can ditch Matte's rules and score to the strongest CQ on either side of the RQ, and you can use +/-6 cut-offs. The research shows that results in better accuracy - the opposite of what you state, which is another built-in bias against the truthful. That aside, you seem to think the Utah test will result in more false positives than the Matte rules, which is interesting given Matte's use of such rules as the "Dual-Equal Strong Reaction" resulting in a negative score. (Frankly, it that doesn't show a bias, I don't know what does.)

Let me get this straight… An examinee shows a maximum reaction (a “three”) on a relevant and an equal reaction on a control. That means the control is defective, or the examinee is screwing with the test. Still, for each of two applicable tracings -- pneumo and cardio only -- a score of -1 is assigned. Are you saying that’s draconian? Remember, the examinee displayed a *maximum* reaction (3) to the relevant question.

>>I was trained in the Matte tests by Jim Matte. For the life of me I can't figure out why he believes his tests have a built-in CM safeguard. To date, he hasn't been able to convince me, and I am not alone there.

I, too, was trained by Jim. Maybe you should take a closer look at what he says in his book. Meanwhile, by all means, identify the other nay-sayers. Name the names.

>>How does the inside track help you assess a false positive? On what basis can you make that claim? If you get a positive score on the inside track, and a negative on the two RQs, do you now presume a false positive?

In such a case, a careful evaluation of the chart is called for. There has to be a reason for that positive score on the inside (fear/hope) track. (Consult the reaction guide in Jim’s book for details regarding evaluation and remedy.)

>>Matte's "seminal study"? How about the only study? That's not a plus. One of the criticisms of the NAS report is how they chose their studies. Many do not consider his study to be high-quality.

What are the weaknesses in Jim’s study? I have a copy of the complete work, so please cite the chapter and verse. And tell us the names of the “many” who consider Matte’s study to be lacking.

>>I will say this though: I am and always have been suspicious of the study. All those cases and no errors? What are the chances?

Are you saying Matte fudged the results? Look, I see your point, but the 100% outcome is possible. Ansley’s meta-study, the one sold by the APA, contains several individual field studies -- including the Matte/Reuss study with the 100% outcome -- claiming accuracy in the high 90s. What’s a few more points? Are you saying a study with 100% accuracy can’t be done? Do you have similar gripes with studies claiming 96%-98% accuracy? If not, why not?

>>Why have there been no independent studies since Jim's?

I don’t know, but two theories come immediately to mind: 1) It would be an extremely labor-intensive project with little payoff for the person doing all that work; 2) In my view, the MQTZCT is not a technique for the average examiner. It’s not popular in large part because it is complex and the proper set-up of the inside track is critical. Some examiners would prefer the “dumbed-down” approach of Utah, especially if it’s agency-endorsed. Makes life easier.

>>How can you and he draw such far-reaching conclusions based on a single study done by the person who has an interest in the outcome?

Speaking for myself, because Jim Matte is an honorable man with a true interest in protecting people from being victimized by false-positive results. Tell us, how do you protect your examinees from false-positive results in your tests?

>> I run my tests as if they will be used in court, which is problematic with this one.

So do I. Do you honestly think that a jury, given the (fully explained) options, would buy into a fixed-threshold scoring system as opposed to an increasing-threshold system? I know you’re wedded to the “data,” but there’s an appreciable element of intuitive appeal to be reckoned with. The data may be black and white, but often the courtroom is simply shades of gray. Hell, even a first-year law school dropout like me could derail the Utah express in cross-examination focusing on a marginal score over five charts.

>>The Utah test and the Utah scoring system are the most researched and supported out there. (Why they aren't more popular is beyond me.) It is the only test that can be used for paired-testing according to ASTM standards as it achieves 90% accuracy.

I’ve observed a lot of name dropping in the polygraph field: “ASTM,” Johns Hopkins University,” “University of Utah,” “Applied Physics Laboratory,” to name a few. It all looks great on a report, but should it come to court, I would put my faith in a jury’s understanding of the inherent logic of an increasing-threshold ZCT administered with validated testing principles.

>>There are other tests out there that will meet the needs of many examiners, I know, but I wouldn't put a lot of faith in thinking the Matte test is our needed salvation.

I merely said, “perhaps it’s time to become re-acquainted with the Matte Quadri-Track ZCT Technique and its underlying principles.” Were I in your shoes, with people’s futures so routinely in your hands, I’d think again. But you’re right -- examiners who so nonchalantly embrace marginal scores in five-chart fixed-threshold exams may need much more than Matte for salvation.

To borrow from a fellow poster, “But that’s just one man’s opinion.”

Okay, now I gotta go refill those ice-cube trays…

posted 11-09-2006 11:45 AM            

There's more here than I have time to get to, but here goes:

quote:

---

Regarding Matte’s Quadri-Track ZCT, Don Krapohl himself told me: “My instincts tell me that it is just as good, but no better, than the other single-issue ZCTs.” Okay, as a relative newcomer (two years in practice), who am I to argue? But personally, I think it’s somewhat “better.” Maybe it’s the asymmetrical, increasing-threshold format coupled with scoring to the preceding control. Maybe it’s the insight afforded by the “inside” (fear/hope) track. Maybe it’s the robust (-15/+9) cut-offs. In any case, after administering a MQTZCT, when I give my report to an attorney, spouse, employer or law enforcement official, I’m highly confident of the result.

---

I agree with Don. It likely is "just as good, but no better...." The Utah and DoDPI research support that conclusion. That is, since the Matte Quadri-Track uses validated principles, then is should work. How well, is another question.

You need to understand that there are two issues here: one involves the test format itself, the second the scoring system used.

Since Matte uses a ZCT with at least a couple good RQs and a couple good CQs, then the CQT research supports that it should work as we expect it to work.

The question that follows is whether the scoring system is optimal. This is not a research project. It is a conversation. If you really want to know what's optimal, read both sides on what works and what doesn't. Rather than debate problems science has already settled, why not forge ahead?

Anyhow, Matte teaches Backster scoring, plus his added rules. Both have too many. What research has shown (and Ray has explained the "whys" elsewhere) is that complex scoring criteria reduces accuracy.

quote:

---

>>First of all, what evidence do you have to show the Backster system would fair this person any better? You're comparing apples to oranges. You want to score Backster features vs. Utah features, and then you want to use Utah cut-offs vs. Backster cut-offs.

Not quite. I was talking about the Matte system, not the Backster system. My apologies if that wasn’t clear. But in any event, both scenarios I described would have resulted in “inconclusive” findings -- with the inside track pointing toward a false positive -- which is vastly preferable to an innocent person being branded a liar. A false positive is what would have happened, in my hypotheticals, with the broad-brush approach inherent to a fixed-threshold scoring system in a case with a marginal score achieved over five charts.

---

Again, Matte and Backster are essentially the same system, but more on that later.

You set up a straw man, and then you knocked him down. Even so, how do you know the "inside-track" would have brought the score up enough to remedy the "problem" of the inconclusive? You don't. You suspect it would, but you have no way of knowing. You are assuming that track can't fail - a stretch from any perspective. You'll notice the NAS apparently didn't think Matte's test fixed that problem - one they had serious concerns about.

Errors are going to occur, and we've got to realize that. You seem to imply that's been fixed with the Matte test, but there's little evidence that is the case, and the polygraph community hasn't jumped on board to your line of thinking. Why do you think that is?

quote:

---

We all know that a competent examiner can detect deception/truth at levels far above chance, but it’s those examinees that fall into the false-positive category – the people that you so easily dismiss with the words “…a certain percentage of the tests will be wrong. We already know that.” – that troubles me so greatly. Matte’s Quadri-Track ZCT was designed specifically to help protect against false positives. If there’s such great ballyhoo about false positives in the court of public opinion, why not take a closer look at a technique designed to minimize such consequences?

---

We don't know that Matte's test does that or not. Let's look at the POLYGRAPH (1989, volume 18, #4) version of his study, which many here will have access to with more ease.

Page 196 tells us that the fear of error question increased scores 7.3 points. The average without that zone was 5.89. In other words, scored as a Bi-spot test (without the inside track zone) the average score was above that (4) needed to conclude an NDI, so what did it add? The same is true with the DI (same page). The increases only come when you use Matte's scoring rules. In other words, he seems to have "optimized" his scoring rules in an attempt to bolster his claim of what a great test he's devised. If he'd stuck to a bi-zone (or spot), he'd have made the same calls. That doesn't do much to support his hypothesis.

quote:

---

I’m interested in knowing what your response to the victim of a false positive result would be in the case of a -6 achieved over five charts. “Oops?” “Stuff happens?” “Sorry about that?” “The system I used is shown to be 90% accurate and is endorsed by leading government researchers, but collateral damage is simply a fact of life in this business?” In your pre-test phase, do you tell your examinees that even though your technique (Utah) is “the most researched and supported out there,” there’s still a solid one-in-ten chance that you’ll be flat-out wrong? Do you make it clear it’s the legal/social equivalent of placing a 10-shot revolver, with one round in the cylinder – and let’s make it a .500 S&W loaded with Golden Dot hollow points for better “dramatics” -- to their head, spinning the chamber and pulling the trigger? Now, you can argue that the Utah system is competitive, on a diagnostic level, with MRI, mammography, EEG, or whatever. But there’s one big difference: Should a patient “fail” one of those medical diagnostic procedures, they likely won’t be charged with a crime.

---

The only honest thing we can do is say that, and yes, I tell people the test isn't perfect. That is different than what Matte suggests when he provides copies of his study to examinees, suggesting perfection, which should eliminate a fear of error should it not? If not, then why? Does that mean your pre-test doesn't do anything?

Who gets charged because of a failed polygraph test? I've got people out of jail with polygraph, but I've never put them in as a result.

You seem to think you are better than 90%. You might be, but how do you know that? As I said, there are two issues here: the test and the scorer. We know (according to the NAS), on average, all CQTs (Matte included)are 86% accurate, but that includes examiner decisions - not what the data actually showed. (FYI: The Utah research showed a 95.5% accuracy in their studies. Don looked at a few studies that met his criteria (which I like), and he came out with a 91% average accuracy rate.)

In the study Don and I did on scoring, we found people are lousy at identifying the NDI - a real problem in my book. However, it is clear the data was there if people knew what to look for as some examiners did very well (i.e., they found them). The test wasn't the problem: the scorer was. Scorers got more symmetrical results with evidentiary decision rules (asymmetrical cut offs that are significantly lower than Mattes). One would expect the wider INC zone to produce just that: more INCs.

quote:

---

Let me get this straight… An examinee shows a maximum reaction (a “three”) on a relevant and an equal reaction on a control. That means the control is defective, or the examinee is screwing with the test. Still, for each of two applicable tracings -- pneumo and cardio only -- a score of -1 is assigned. Are you saying that’s draconian? Remember, the examinee displayed a *maximum* reaction (3) to the relevant question.

---

How do you know it means the examinee is lying to the RQ or is messing with you? You assume the anti-climax dampening concept really holds true in the real world, but it doesn't (and I was taught that in polygraph school too). We do know that the innocent don't react to CQs as much as the DI react to RQs. We also know - as you point out - that a fear of error could account for reactions to the RQs. If that's the case, then why do you bias your scoring against the truthful by giving a negative score to to equally strong reactions? The fix, you say, (according to Matte's chart) is to reduce the intensity of the CQ. How is that not a bias against the truthful?

quote:

---

Are you saying Matte fudged the results? Look, I see your point, but the 100% outcome is possible. Ansley’s meta-study, the one sold by the APA, contains several individual field studies -- including the Matte/Reuss study with the 100% outcome -- claiming accuracy in the high 90s. What’s a few more points? Are you saying a study with 100% accuracy can’t be done? Do you have similar gripes with studies claiming 96%-98% accuracy? If not, why not?

---

I'm saying what I said. I have a problem with the 98% accuracy studies when most examiners (half according to the NAS, which was consistent with what Don and I found) can't get past the 86% mark. If memory serves me, Matte did another "study" on the Backster test and wow, another 100% accuracy.

quote:

---

It’s not popular in large part because it is complex and the proper set-up of the inside track is critical. Some examiners would prefer the “dumbed-down” approach of Utah, especially if it’s agency-endorsed. Makes life easier.

---

Nonsense. It's no more difficult than any other.

More later. I've got to run.

posted 11-09-2006 03:27 PM            

quote:

---

Speaking for myself, because Jim Matte is an honorable man with a true interest in protecting people from being victimized by false-positive results. Tell us, how do you protect your examinees from false-positive results in your tests?

---

That's it? He's well meaning, so he must be right? Is that really what you meant? Does that mean others aren't interested in protecting from false positives? That's insulting - and way off base.

You have a test that purports to measure "fear" something we know is not necessary for polygraph to work. It's an extra issue - if it doesn't do what it's supposed to do - which we all know reduces accuracy. Is is a bad test? No, I don't think so. Is it the panacea Jim tries to make it out to be? Again, I don't think so.

I protect people from a false positive by running a good test based on science - not opinion and anecdotal experience tainted by our own biases. That's where this profession needs to be.

quote:

---

Do you honestly think that a jury, given the (fully explained) options, would buy into a fixed-threshold scoring system as opposed to an increasing-threshold system? I know you’re wedded to the “data,” but there’s an appreciable element of intuitive appeal to be reckoned with. The data may be black and white, but often the courtroom is simply shades of gray. Hell, even a first-year law school dropout like me could derail the Utah express in cross-examination focusing on a marginal score over five charts.

---

You don't have much experience in the courtroom do you? You want to testify about your opinion before you establish the science behind it? It won't happen that way. You don't get polygraph to a jury until you get past the judge, who's going to have a Daubert hearing first. Sure, you could come in and muddy the waters by talking about an unvalidated scoring method, but who knows how that would turn out?

You'd still have the problem of the data. The Utah research supports the +/-6 as very accurate. According to our research - as I said before - a +4/-6 is optimal, so I do agree with asymmetrical cut-off scores. Changing scores will only change errors though - and you can't forget that fact. Again, scoring vs. format equals apples and oranges.

Whatever you think you could derail, you don't won't have the data to support the very high cut-offs Matte uses. Moreover, scoring to the preceding CQ produces a bias against the truthful when the RQ is bracketed by CQs. Think about it: when you score to the strongest CQ, scores will go up (if there is any change).

In the end, I can argue that I used a validated scoring system to score my test (even a Matte test), and you couldn't. Do you see where this is going? It creates an in-house debate over polygraph. The anti crowds don't even have to do anything but sit back and watch.

We need to be careful as to what we preach. We really have nowhere to go but to the data. All else is speculation, and that can produce a greater risk of what you seek to avoid.

quote:

---

I merely said, “perhaps it’s time to become re-acquainted with the Matte Quadri-Track ZCT Technique and its underlying principles.” Were I in your shoes, with people’s futures so routinely in your hands, I’d think again. But you’re right -- examiners who so nonchalantly embrace marginal scores in five-chart fixed-threshold exams may need much more than Matte for salvation.

---

What is "nonchalant" about using science to achieve optimal results? You have no idea what you're using. You have one study - a study that determined ground truth based on the police investigation in a large percentage of the cases. (I wonder how much any of those investigations were influenced by polygraph results?)

Additionally, isn't it "nonchalant" to use a scoring system with unknown accuracy? (Like Don, I'm guessing it'll work most of the time, but why guess?) How does the Utah system use "marginal" cut-offs? You base that on your assumption that Matte is right, which means they must be wrong. (Several studies verses one, I might point out again.)

You seem to imply a police test is somehow inferior to what a private test would do for a person. Anecdotally, I've experienced the opposite.

That is exactly why I use the Utah test. I know what it can do, and I know what I can do as a scorer.

There is nothing "nonchalant" about a five-chart test. It only reduces inconclusives - it doesn't create more FPs as you suggest in your straw man example. (That's what the data says anyhow.)

The principles underlying the Matte test have, in some instances, been disproved, but the test remains the same. Why?

Which is better, running a Backster test with the inside track or the Matte test with the inside track? Which scoring system is better? What cut-offs are better. Why? How do you know? (Matte teaches that it can be run in both.)

If I haven't been clear, there are two issues I want to emphasize: 1) run a good, validated test (based on validated principles), and 2) use a validated scoring system.

posted 11-09-2006 07:55 PM         

You've given me much on which to cogitate. I appreciate that, along with your candor. I'm sorry if any of my comments offended you or anyone else in the polygraph community. Such was not my intent; I'm just an FNG who only wanted to engage in a spirited discussion.

Now I'm wondering... Given everything that's been said, do you think the polygraph community is headed toward -- for lack of a better phrase -- a "Utah-centric new world order?" If so, what would that spell for the old-line schools (such as my alma mater, Backster) that don't embrace the Utah technique? Would their APA accreditation be in jeopardy at some point? I realize nothing is going to happen overnight, but what do you hold for Utah's influence over the long haul?

I'd really like to hear what anyone has to say about the evolutionary process. Any theories?

posted 11-09-2006 08:57 PM            

quote:

---

Now I'm wondering... Given everything that's been said, do you think the polygraph community is headed toward -- for lack of a better phrase -- a "Utah-centric new world order?" If so, what would that spell for the old-line schools (such as my alma mater, Backster) that don't embrace the Utah technique? Would their APA accreditation be in jeopardy at some point? I realize nothing is going to happen overnight, but what do you hold for Utah's influence over the long haul?

---

Look, the Utah researchers would be the first to tell you that their research supports the concept that any well put together CQT is valid, so Backster's not going to disappear. I think Lou Rovner said it best in his most recent article in the APA magazine, which I'll paraphrase: When you learn a better way to do something, you have a duty to do it.

Backster's formats are fine. His methods of introducing CQs and phrasing them aren't optimal, and his method of scoring isn't either. In fairness, Jim Matte argues Backster puts more emphasis on introducing them than his critics claim, but I haven't seen that in the very few Backster cases I've seen. So, yes, I think you will see a change to more scientific (i.e., defendable) methods, but don't forget DoDPI is also conducting a lot of research too. Some of Utah's "stuff" is based on DodPI's. (And some of DoDPI's is based on Backster's. Without Cleve and Dick, there'd be no Utah or DoDPI).

I have found there is a fight in the polygraph community between what is right and what is convenient. I have a real ethical problem with that, but I think right will prevail. How long it will take, I don't know. There is also a problem with egos, and those will never go away, but let's not forget that pride comes before a fall.

I hope there is a day when our schools are really required to teach "validated" techniques. They're supposed to now, but we've never really defined "validated." Don just made that attempt in his most recent article. Essentially, we've said that if an APA school teaches a particular technique, then it's "valid" - even when it isn't any good (like the AF MGQT, or "the DI test" as they called it in polygraph school).

Wasn't Utah's Charles Honts, one of the most prolific polygraph authors, a Backster grad? It was scientific testing of what he learned there that got him where he is today.

In the end, let's not leave this looking like the Matte test is bad. It isn't, and quite frankly, if it does what Matte claims (and I don't know if it does), then it's a great test for a breakout on a screening test that went SR. Think about it: a guy with a job on the line just got told he reacted significantly to a question he's telling the truth on. He's really got a fear of error now. (I have used the test in that circumstance. I just don't score it like Matte would because his ways make a false positives more likely than the scoring methods I use.)

posted 11-09-2006 09:45 PM         

Now boys, this is EXACTLY what this site is for! I would love to ring in here but I have to run....my daughter just went into labor( Really!).

Keep the discussion going...I'll be back.

posted 11-10-2006 02:40 AM            

To your corners gentlemen.

Sorry I missed all this today.

I've been in the mountains dodging deer - it seems they hide on the highways during hunting season.

I've a lot to say, but much of it would be repetitious.

The short version of my thoughts goes like this...

Neither the Utah or Backster/Matte systems are perfect.

If we thought the inside track solved all false-positive or CM problems perfectly (or even reliably), then someone would be accepting Drew Richardson's challenge... For that matter someone would probably accept the challenge if we thought that our motion sensors and confidence were all that mattered.

One of the challenges of this type of discussion is to cite our sources adequately while using hypotheticals and metaphors to describe things that are possible - without creating the impression that those hypotheticals and metaphors are themselves definitive. So thanks people for citing sources.

One of my pet peeves, from sitting in too many state committee meetings with too many myopic agendas, is the statement "research says..." without an adequate cite.

Another pet peeve has to do with professionals who replace empirical confidence with their personal confidence. Empirical confidence is based on data, measurement, and math. Personal confidence is based on personality, charisma, persuasiveness, stubbornness, and forcefulness. Science is sometimes unfortunately fraught with uncertainty and other complications.

The real challenge is to move toward techniques and principles that are going to be increasingly describable in the common languages of the sciences of psychology, physiology, measurement, testing, and inferential statistics. These are among the requirements underlying what it means to be "validated." Validation studies are also necessary, but if they describe the polygraph in idiosyncratic terms that are inconsistent with generally accepted empirical principles, then we've missed the point.

While I've been playing with significance tests, I'm finding that the field practice guidelines that are emerging from recent research - such as two stage decision rules - can be quite consistent with the commonly accepted principles in significance testing. Here is an example.
http://www.raymondnelson.us/qc/split_061109.pdf

a recent maintenance polygraph which resulted in scores of +4 0 and -3 to the three RQs.

You can see that both the parametric and non-parametric algorithms parsed the questions to a DI test - because the -3 (for this dataset) met the Bonferonni corrected alpha.

I didn't set the -3 threshold to be DI. In significance testing, threshold are set by alpha levels. On tests with few scores that deviant from zero, results of +/- 2 can be significant. On other tests, with lots of robust numbers, +/- 3 may not be significant.

Does that sound like increasing thresholds??? The idea is not without merit. However, it is probably a mistake to try to mimic the actions of statistical procedures, without actually doing the math.

I think a lot of smart people can see the point about asymmetrical decision thresholds, and this is reflected in some of the newer research based systems such as OSS.

However, the issue surrounding increasing thresholds begs more discussion. One of the generally accepted principles of testing is that all test scores or measurements are estimates of the true score. IQ test results are not IQs, they are estimates of IQs. As such, they are reported as an estimate and a confidence interval - usually 90 percent or 95 percent. What confidence intervals mean is that we are X% confident that if we tested the individual on any given day, the score would fall within that range. Confidence intervals are calculated using standard errors - a common statistical method, that evaluates the variability within a sample, and the size of the sample.

Because scores and measurements are themselves estimates, we don't use just one. We take several measurements and scores, with the understanding that every measurement/score is a reflection of the universe score (sometimes called the "true" score, but that provokes assumptions about the word "true"), and the ever present and unavoidable influence of random measurement error.

Measurement error in polygraph is a feature of physiological and psychological variability at the time of testing, component sensor efficiency, and measurement efficiency. Of these, measurement efficiency is the thing we have the most direct control over, and is the source of all these differences of opinion in what constitutes a scoreable reaction.

Another generally accepted principle in testing sciences is that tests and measurements can be no more accurate than they are reliable - if you can get the same answer each time, then you really don't know what you are measuring. So, the issue of scoring systems and reaction features is a critical one. The way to achieve a truly reliable measurement is to mechanize it, which is what the Utah and Kircher features have attempted to do. Mechanized measurements provide the opportunity for machine/computer measurements, and machines can be expected to have near perfect reliability regardless of complexity (humans get tired, and procedures drift - machines don't). OSS is based on mechanical measurements, and that in part underlies its strength and validity.

An example of a data feature that has not yet mechanized is our countermeasure data.

Now back to my point about increasing thresholds and fixed thresholds. What we're really talking about is the role of variability in our scored data. If every measurement score is itself a true or universe score, then adding those up to an increasing threshold is entirely reasonable. But what if every score is itself an estimate, with variability and error (anyone want to argue that they are not)? One of the ways that we stabilize variability and error is by using multiple measurements, and let the variability wash out in averages, deviations, and distribution shapes. We all seen tests with a mixture of + and - scores to questions. It is the aggregate that we are concerned about - it would be unsound to attempt to render an opinion based on a single question or single chart. What we look for is consistency and trends - or more correctly, the shape and location of the distributions of scores.

In fact, without some variability, some of our favorite statistical methods don't work. Look at this hypothetical example of what happens to the parametric statistics when a test results in perfectly consistent scores.
http://www.raymondnelson.us/qc/hypothperfectdi1.pdf

The results work, and the data satisfy a Kolmogorov-Smirnov test for normalcy (meaning the distribution shape resembles a normal bell-curve). However, normalcy tests are somewhat limited and Q-Q plots can be more informative - look at the quantile plots. Normally distributed data will plot a 45 degree angle on quantile plots. These data are not even close, which is no surprise, because there is no variability. This is just a good example of why variability is important. Look at the same data with one changed value.
http://www.raymondnelson.us/qc/hypothperfectdi2.pdf

The Q-Q plots are substantially more normal looking. The first example would not justify the use of our favorite parametric algorithms, the second example would.

As for some research, I suppose I'll offer to help, but we're gonna need some data. Plus, someone should start to clarify the question ahead of time.

----------

Aside from all that congratulations Ted. I hope all goes well.

peace,

posted 11-10-2006 06:56 AM            

I read your article, and I liked it. I just wrote a chapter on the Utah test for the new AAPP handbook, which won't be out until January at the earliest. I had it reviewed by the same folks, so it looks like we're reading from the same play book.

Why not point out the deviations here for starters? That'll get the conversation going a little faster. Besides, since we've used the same people for review, it's probably a poor explanation on my part that you don't like.

Ray,

I hate to see articles with no cites too. Here though, it's not a problem unless it really becomes necessary. I presume I'm talking with people who have read the literature and are somewhat familiar with it. If we cited the research for everything every time, all responses would be research projects, and who has time for that?

Bill,

posted 11-10-2006 12:53 PM         

People, I want to emphasize that my overarching concern is the protection of innocents from being victims of false-positive results. I recognize the appeal of "machination" relative to reliability and validity. Perhaps, someday, we'll get to a point where we're all comfortable. For now, it seems to me that the real-world dilemma we face is teasingly reflected in the title of Matte's first book: "The Art and Science of the Polygraph Technique." Kind of says it all, doesn't it?

posted 11-10-2006 02:25 PM            

I agree with you, and I am concerned about the false positive problem. But, and that's a big but, some use polygraph as an investigative tool where their fear is a false negative, as false positives have little cost in those contexts (since people are going to be charged with crimes based on polygraph results).

A false negative could slow or end an investigation, and the real person is overlooked. Because that is considered too big of a price to pay, they run tests designed to make it harder for people to pass. The cost is more false positives.

The cost of reducing false positives, is of course, the opposite, more false negatives.

Polygraph is only so accurate, and all we can do is shift our errors (and / or widen the INC zone) in one direction or the other. What is acceptable is a matter of personal or agency philosophy.

That is why we now have Investigative Scoring Rules (for situations in which false negatives are a concern) and Evidentiary Scoring Rules (for situations in which a balanced accuracy is needed, e.g. the court room).

I've said this again and again: Backster (and Matte and the Virginia school, etc.) was right about his asymmetrical cut-offs, but Utah and DoDPI are "better" in that they have made improvements and made scoring more standardized.

All others,

I am not up for writing a research paper every time I write here; however, what are your thoughts about creating a bibliography of sorts in which we catalog studies and a one or two sentence blurb on what the study concluded? We could add those studies that interest us here (and we could talk about any we want), and in the end we could end up with a nice doc (that Ray could post?) with a lot of info on it. It's a major project, as there's a lot out there, but in the end, we've have a great reference that we could build on from here on out.

Here's an example:

Krapohl, D., Stern, B., and Bronkema, Y. (2003). Numerical Evaluation and Wise Decisions. Polygraph, 32(1), 2-14.

Discusses validated scoring features with supporting cites along with a discussion of validity, reliability and variance.

In the end, we'd have hundreds of docs.

I'm willing to compile them if we create a thread to post them. I'll just cut-and-paste them into a Word doc that just keeps expanding.

posted 11-10-2006 02:30 PM            

quote:

---

I want to emphasize that my overarching concern is the protection of innocents from being victims of false-positive results. I recognize the appeal of "machination" relative to reliability and validity. Perhaps, someday, we'll get to a point where we're all comfortable. For now, it seems to me that the real-world dilemma we face is teasingly reflected in the title of Matte's first book: "The Art and Science of the Polygraph Technique." Kind of says it all, doesn't it?

---

Lets move this to a different thread. Its a hugely important topic, and has as much to do with policy matters and what people do with test as it does with science and what test results are capable of meaning.

(I got the .pdf-s up so you can see the plots, if you are interested).

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 11-10-2006 04:17 PM            

Another minor diffrence (and it may be simply semantics) is the decision criteria for multiple facted examinations. Dr. Honts taught me to use the Grand Total first as long as all spot scores are in the same direction. As long as the Grand Total exceeded the +/-6 you make a call of DI or NDI. Only if you have spot scores in opposite directions do you resort to the spot score of +/-3. I may have misread your explaination in the paper in Raymond's posting.

Additionally I was taught one could only assign one 3 per chart in the EDA and cardio channels and only if the tracings were stabile. If there was lability and NSRs one is limited to a +/-2.

posted 11-10-2006 05:29 PM            

Anyhow, there you have it. Change it to three, or better yet, start eyeballing RLL as it's pretty easy to see what will ultimately result in a scoreable ratio.